Approximate Dynamic Programming Research Articles

Approximate Dynamic Programming for Control of Wave Energy Converters With Implementation and Validation on a Point Absorber Prototype

It is recognized that the application of constrained optimal control for wave energy converters (WECs), represented by model predictive control (MPC), is hindered by its computation burden resulting from online optimization, especially when the model contains nonlinearities. In this article, a novel control solution based on approximate dynamic programming (ADP) is proposed, in which an explicit controller is solved by model sampling and neural network training. In this way, the major computation load is moved offline to enable fast online execution, and the need for wave prediction is eliminated. The effectiveness of ADP is verified by a simulation case study based on a nonlinear WEC system. Using nonlinear MPC as the performance benchmark, over 90% energy efficiency is achieved by ADP under various sea states with constraint satisfaction and dramatically faster computation. The ADP controller is further experimentally validated on a newly constructed WEC prototype through wave tank testing, where successful implementation and satisfactory performance are demonstrated.

Adaptive network traffic control with approximate dynamic programming based on a non-homogeneous Poisson demand model

ABSTRACT In this study, we develop a stochastic dynamic traffic-flow model subject to practical restrictions under the non-homogeneous Poisson vehicle arrival process. Using the cell transmission strategy, we establish traffic dynamics between two intersections. We also discuss simulating the random demand of source links from an estimated intensity function. Additionally, we propose an algorithm to optimize time interval division for aggregated data, aiming to enhance estimation performance. We explore applying our traffic flow model to the adaptive traffic network management problem, which is formulated as a Markov decision process. Leveraging approximate dynamic programming with recursive least squares-temporal difference learning, we achieve adaptive optimal policies. To validate our approach, we conduct a series of numerical experiments with random demands. The results of non-homogeneous Poisson demand conducted using random numbers and a real-word dataset indicate high efficiency with the piecewise constant, I-SMOOTH, and MNO-PQRS estimators. Compared to the Webster and Max-pressure control systems, our proposed approximate dynamic programming-based model exhibits superior stability and applicability.

Policy iteration for H∞ control of polynomial time‐varying systems

AbstractThis paper studies the H∞ control problem for polynomial time‐varying systems. The H∞ control problem has been much less investigated for time‐varying systems in comparison to the time‐invariant systems. Approximate dynamic programming (ADP) is an optimal method to solve the control problems. Therefore, it is valuable to solve the polynomial time‐varying H∞ control problem with the ADP approach. Considering the time as an independent variable for sum‐of‐squares (SOS) optimization problems, an SOS‐based ADP method is proposed to solve this problem. A policy iteration algorithm is presented, where in its policy evaluation step it is sufficient to solve an optimization problem. Some constraints are added to this optimization problem to guarantee the closed‐loop exponential stability. The convergence and stability properties of the proposed algorithm are stated and proven. Moreover, in order to design an H∞ controller with a smaller disturbance attenuation coefficient, a two‐loop algorithm is suggested. Finally, the effectiveness of the proposed method is demonstrated by simulation examples.

Approximate optimal and safe coordination of nonlinear second-order multirobot systems with model uncertainties

This paper investigates the approximate optimal coordination for nonlinear uncertain second-order multi-robot systems with guaranteed safety (collision avoidance) Through constructing novel local error signals, the collision-free control objective is formulated into an coordination optimization problem for nominal multi-robot systems. Based on approximate dynamic programming technique, the optimal value functions and control policies are learned by simplified critic-only neural networks (NNs). Then, the approximated optimal controllers are redesigned using adaptive law to handle the effects of robots’ uncertain dynamics. It is shown that the NN weights estimation errors are uniformly ultimately bounded under proper conditions, and safe coordination of multiple robots can be achieved regardless of model uncertainties. Numerical simulations finally illustrate the effectiveness of the proposed controller.

Branch-and-Price for Prescriptive Contagion Analytics

A new optimization methodology helps allocate vaccines to combat a pandemic. During the COVID-19 pandemic, the extraordinary speed of vaccine developments quickly gave rise to a critical operational problem: How to distribute a scarce stockpile of vaccines across communities? In “Branch-and-Price for Prescriptive Contagion Analytics,” A. Jacquillat, M. Li, M. Rame, and K. Wang address this question by presenting a methodology to allocate shared resources across subpopulations governed by contagion dynamics. This problem combines the difficulties of mixed-integer nonconvex optimization and those of optimization with constraints governed by ordinary differential equations. By combining novel column generation, approximate dynamic programming, and branch-and-bound elements, the authors’ methodology can solve large and otherwise-intractable instances, outperforming state-of-the-art benchmarks. From a practical standpoint, their approach can significantly enhance the effectiveness of a vaccination campaign. Ultimately, the results from the paper outline vaccine allocation—beyond vaccine development and vaccine manufacturing—as a critical lever to mitigate the impact of a pandemic on public health.

Technician routing and scheduling with employees’ learning through implicit cross-training strategy

With record high talent shortages and skill mismatches around the world, this paper investigates a variant of multi-period dynamic technician and routing problem that can be modeled as a Markov decision process. To deal with the double tradeoffs between the routing and service time costs, as well as the current and future costs, we propose an approximate dynamic programming (ADP)-based cost function approximation (CFA) algorithm — the implicit cross-training strategy (ICT). A two-phase routing and scheduling heuristic is developed to account for both employees’ learning and future information, and to facilitate an efficient implementation of CFA. Extensive computational results show that ICT can provide a better solution in the current decision with a global view in comparison with the myopic strategy. In depth analysis demonstrates that ICT trains the workforce with more balanced skillsets and workloads, which ensures the flexibility of the workforce and helps buffer against the future uncertainties with substantial routing cost savings. Additionally, ICT has much more advantages in large-scale problems with more diversified service requests and randomly distributed customers.

Data-Efficient Off-Policy Learning for Distributed Optimal Tracking Control of HMAS With Unidentified Exosystem Dynamics.

In this article, a data-efficient off-policy reinforcement learning (RL) approach is proposed for distributed output tracking control of heterogeneous multiagent systems (HMASs) using approximate dynamic programming (ADP). Different from existing results that the kinematic model of the exosystem is addressable to partial or all agents, the dynamics of the exosystem are assumed to be completely unknown for all agents in this article. To solve this difficulty, an identifiable algorithm using the experience-replay method is designed for each agent to identify the system matrices of the novel reference model instead of the original exosystem. Then, an output-based distributed adaptive output observer is proposed to provide the estimations of the leader, and the proposed observer not only has a low dimension and less data transmission among agents but also is implemented in a fully distributed way. Besides, a data-efficient RL algorithm is given to design the optimal controller offline along with the system trajectories without solving output regulator equations. An ADP approach is developed to iteratively solve game algebraic Riccati equations (GAREs) using online information of state and input in an online way, which relaxes the requirement of knowing prior knowledge of agents' system matrices in an offline way. Finally, a numerical example is provided to verify the effectiveness of theoretical analysis.

A kernel-based approximate dynamic programming approach: Theory and application

This article proposes a novel kernel-based Dynamic Programming (DP) approximation method to tackle the typical curse of dimensionality of stochastic DP problems over the finite time horizon. Such a method utilizes kernel functions in combination with Support Vector Machine (SVM) regression to determine an approximate cost function for the entire state space of the underlying Markov Decision Process (MDP), by leveraging cost function computed for selected representative states. Kernel functions are used to define the so-called kernel matrix, while the parameter vector of the given kernel-based cost function approximation is computed by moving backwards in time from the terminal condition and by applying SVM regression. This way, the difficulty of selecting a proper set of features is also tackled. The proposed method is then extended to the infinite time horizon case. To show the effectiveness of the proposed approach, the resulting Recursive Residual Approximate Dynamic Programming (RR-ADP) algorithm is applied to the sensor scheduling design in multi-process remote state estimation problems.

Biobjective optimization for railway alignment fine‐grained designs with parallel existing railways

AbstractUrban high‐speed railway construction is complex due to limited land resources, high population density, and potential construction risks, especially when new tracks are parallelly aligned to operational railways. Addressing a gap in current literature on fine optimization of manual alignment in such scenarios, this paper introduces a biobjective approximate fine‐grained optimization model for railway alignments (BA‐FORA). Utilizing an approximate dynamic programming (ADP) method, BA‐FORA effectively searches the feasible region to approach a global optimum, overcoming the dimensionality challenges inherent in standard dynamic programming (DP). This paper presents a biobjective optimization framework that takes into account both construction cost and construction risk adjacent to existing operating railways (CRAEOR), offering a method for the fine‐grained design of new railways adjacent to existing railways. Finally, the proposed BA‐FORA framework is applied to practical cases, demonstrating its superior optimization performance. The findings indicate that the BA‐FORA model can autonomously investigate and enhance railway alignment. It generates cost‐effective and low‐risk solutions exceeding manual efforts, ensuring alignment constraint compliance.

Robust optimal control of the multi‐input systems with unknown disturbance based on adaptive integral reinforcement learning Q‐function

AbstractConsidering overshoot and chatter caused by the unknown interference, this article studies the adaptive robust optimal controls of continuous‐time (CT) multi‐input systems with an approximate dynamic programming (ADP) based Q‐function scheme. An adaptive integral reinforcement learning (IRL) scheme is proposed to study the optimal solutions of Q‐functions. First, multi‐input value functions are presented, and Nash equilibrium is analyzed. A complex Hamilton–Jacobi–Issacs (HJI) equation is constructed with the multi‐input system and the zero‐sum‐game‐based value function. It is a challenging task to solve the HJI equation for nonlinear system. Thus, A transformation of the HJI equation is constructed as a Q‐function. The neural network (NN) is applied to learn the solution of the transformed Q‐functions based on the adaptive IRL scheme. Moreover, an error information is added to the Q‐function for the issue of insufficient initial incentives to relax the persistent excitation (PE) condition. Simultaneously, an IRL signal of the critic networks is introduced to study the saddle‐point intractable solution, such that the system drift and NN derivatives in the HJI equation are relaxed. The convergence of weight parameters is proved, and the closed‐loop stability of the multi‐system with the proposed IRL Q‐function scheme is analyzed. Finally, a two‐engine driven F‐16 aircraft plant and a nonlinear system are presented to verify the effectiveness of the proposed adaptive IRL Q‐function scheme.

Event-Triggered Cooperative Tracking Control of Multi-Agent Systems With a Dynamic Leader via Approximate Dynamic Programming

Event-Triggered Cooperative Tracking Control of Multi-Agent Systems With a Dynamic Leader via Approximate Dynamic Programming

A Fully Distributed ADP Algorithm for Real-Time Economic Dispatch of Microgrid

This paper proposes a fully distributed approximate dynamic programming (FD-ADP) algorithm framework for real-time economic dispatch of a microgrid (MG). Firstly, a modified distributed consensus-based algorithm (DCBA) framework with power restriction factors is proposed to distributedly obtain the real-time ED decision of MG containing distributed energy resources with linear cost functions, which improves the convergence stability compared with the traditional DCBA framework. Then, combined with the approximate dynamic programming (ADP) theory, a FD-ADP algorithm is proposed to deal with data uncertainties. To this end, the piece-wise linear (PWL) function is applied to approximate the value function, and a distributed slope update approach is proposed to embed empirical knowledge into the value function while ensuring the distributed property in off-line training. With the well-trained PWL function assisting in decision, the distributed real-time ED strategy of MG with approximate global optimality is obtained by the proposed FD-ADP algorithm during the entire schedule horizon (e.g., 24 hours). Finally, case studies on the modified IEEE 6-bus MG system and IEEE 123-bus MG system demonstrate the superiority of solution convergence, validity of distributed training, and optimality of distributed schedule of the proposed FD-ADP algorithm.

Approximate Dynamic Programming with Enhanced Off-policy Learning for Coordinating Distributed Energy Resources

Approximate Dynamic Programming with Enhanced Off-policy Learning for Coordinating Distributed Energy Resources

Data-Driven Optimal Tracking Control for Discrete-Time Nonlinear Systems With Unknown Dynamics Using Deterministic ADP.

This article aims to solve the optimal tracking problem (OTP) for a class of discrete-time (DT) nonlinear systems with completely unknown dynamics. A novel data-driven deterministic approximate dynamic programming (ADP) algorithm is proposed to solve this kind of problem with only input-output (I/O) data. The proposed algorithm has two advantages compared to existing data-driven deterministic ADP algorithms for the OTP. First, our algorithm can guarantee optimality while achieving better performance in the aspects of time-saving and robustness to data. Second, the near-optimal control policy learned by our algorithm can be implemented without considering expected control and enable the system states to track the user-specified reference signals. Therefore, the tracking performance is guaranteed while simplifying the algorithm implementation. Furthermore, the convergence and stability of the proposed algorithm are strictly proved through theoretical analysis, in which the errors caused by neural networks (NNs) are considered. At the end of this article, the developed algorithm is compared with two representative deterministic ADP algorithms through a numerical example and applied to solve the tracking problem for a two-link robotic manipulator. The simulation results demonstrate the effectiveness and advantages of the developed algorithm.

Robust Approximate Dynamic Programming for Nonlinear Systems With Both Model Error and External Disturbance.

Model error and external disturbance have been separately addressed by optimizing the definite H∞ performance in standard linear H∞ control problems. However, the concurrent handling of both introduces uncertainty and nonconvexity into the H∞ performance, posing a huge challenge for solving nonlinear problems. This article introduces an additional cost function in the augmented Hamilton-Jacobi-Isaacs (HJI) equation of zero-sum games to simultaneously manage the model error and external disturbance in nonlinear robust performance problems. For satisfying the Hamilton-Jacobi inequality in nonlinear robust control theory under all considered model errors, the relationship between the additional cost function and model uncertainty is revealed. A critic online learning algorithm, applying Lyapunov stabilizing terms and historical states to reinforce training stability and achieve persistent learning, is proposed to approximate the solution of the augmented HJI equation. By constructing a joint Lyapunov candidate about the critic weight and system state, both stability and convergence are proved by the second method of Lyapunov. Theoretical results also show that introducing historical data reduces the ultimate bounds of system state and critic error. Three numerical examples are conducted to demonstrate the effectiveness of the proposed method.

A Novel Approximate Dynamic Programming Structure for Optimal Control of Discrete-Time Time-Varying Nonlinear Systems

A Novel Approximate Dynamic Programming Structure for Optimal Control of Discrete-Time Time-Varying Nonlinear Systems

Development and Adaptive Assistance Control of the Robotic Hip Exoskeleton to Improve Gait Symmetry and Restore Normal Gait

Neurological disorders and aging induce abnormal gait kinematics such as asymmetric gait, reduced stride length, and walking speeds. Gait asymmetry increases musculoskeletal pain and joint impairment in the affected leg, increases the risk of patients falling, and induces higher energy expenditure. This paper presents a novel assistive control framework for the series elastic actuator-driven robotic hip exoskeleton that improves gait symmetry and restores normal gait. The proposed assistive control depends on virtual stiffness compensation with a series elastic actuator (SEA)-driven hip exoskeleton to assist without needing estimation, prediction of the gait phase, or human intent. In addition, adaptive adjustment of the virtual stiffness based on the approximate dynamic programming method is used to improve gait symmetry. The proposed control strategy is successfully implemented, and experiments are validated with human subjects’ gait assistance using a SEA-driven wearable hip exoskeleton. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This paper was motivated by the problem of hip exoskeleton assistive control. The goal of hip exoskeleton assistance is to improve gait symmetry and restore the normal gait of the patient with neurological disorders and the elderly. Existing approaches to fixturing such parts generally need to estimate or predict the gait phase or human intent. And the existing approaches generally adopted fixed control parameters. The hip exoskeleton’s assistance cannot adapt to various walking conditions and human gait differences. The proposed assistive control is based on virtual stiffness compensation with a SEA-driven hip exoskeleton to assist with walking under various walking conditions without estimating or predicting the gait phase or human intent. And the proposed method is online adjusted control parameters with human-in-the-loop to improve the gait symmetry. The human walking experiment results showed that the hip exoskeleton assistive controller not only reduced muscle activation and human effort but also improved the interjoint coordination of the impaired joint online. We also showed how the assistance controller changes the control parameters according to the gait symmetry with a human-in-the-loop strategy.

Sequential learning based re-optimization approaches for less model-based dynamic pick-up routing problem

We address a lessmodel-based dynamic routing problem arising from home parcel pick-up service, where lessmodel-based means existing customers who dynamically request services independently following Poisson process with a stochastic rate. Overall, through an extended application of re-optimization (RO) strategy, a Markov decision process formulation and approximation dynamic programming- and Bayes' theorem-based solution approaches are proposed. Specifically, first a pool of basic policies corresponding to all possible values of the rate are developed offline via approximate value iteration. Then, Bayes' theorem-based sequential learning is designed that can sequentially update the belief about the rate's probability distribution over its possible values. Third, coupled with the updated belief, basic policies are collectively implemented in two different ways, resulting in RO approaches which involve constructions of two different online policies, i.e. a belief-weighted deterministic policy and a belief-based random policy, and their re-optimizations at decision epochs. In the numerical study, through comparison with model-based (i.e. using full knowledge of the rate) and model-free heuristics, our approaches are examined and valuable insights are obtained. Important insights include that (i) the belief-weighted deterministic policy outperforms the belief-based random policy, and further, (ii) the former is better than the latter at preserving the improvement resulting from the improved model-based policy.

Survey of Optimization Techniques in Interconnected Two Area Power system for the Application of AGC

The increasing need for energy has placed optimization at the center of power engineering research and development. Presenting the theoretical underpinnings of optimization from a practical power system perspective, this review article on Electric Power System Applications of Optimization explores state-of-the-art techniques, novel approaches, and ongoing application difficulties. This work addresses several algorithmic issues that arise from the planning and management of power systems using diverse approaches, as well as the analytical formulation of optimization. This article introduces new features related to market programs, pricing, reliability, and developments in intelligent systems with working algorithms and examples. This paper discusses the latest developments in the fields of adaptive feedback design and approximate dynamic programming. In order to describe new concepts and variations of Adaptive Dynamic Programming, this work begins with fundamental theories and theorems from functional optimization, optimum control, and dynamic programming. Because of its innovative integration of basic optimization principles with application, this review article encourages power engineers to make new discoveries in the provision of energy at the optimal rate.

Multistage reliability-constrained stochastic planning of diamond distribution network: An approximate dynamic programming approach

In this paper, a multistage reliability-constrained stochastic planning model is established for the diamond distribution network, a new type of distribution system grid structure, with the goal of optimizing the total cost of investment and operation. The uncertainty of load growth is simulated by generating multiple load growth scenarios, and the power supply loss caused by the reliability of components in the system is also considered. The multistage stochastic dynamic decision process is modeled by The Markov decision process. In order to solve the “curse of dimensionality” problem in multi-stage stochastic planning model, the approximate dynamic programming algorithm is used to solve it. The case study is based on the typical structure of the diamond distribution network and realistic data of an actual district in Shanghai, China. The planning results verify the effectiveness of the model and the solution algorithm, and demonstrate the reliability and economy of the diamond distribution network are superior to the traditional double-ring network.